Add DLM force merge operation helpers by seanzatzdev · Pull Request #143200 · elastic/elasticsearch

seanzatzdev · 2026-02-27T04:04:34Z

This PR adds some helpers to perform the actual deduplicated DLM force merge, and removes some extraneous testing per feedback on a previous PR

For the most part, this code just adapts the existing logic from data lifecycle service. there is some duplication due to this in order to avoid further refactors

Copilot

Pull request overview

This PR adds helper methods to perform deduplicated DLM force merge operations within the ForceMergeStep class, and removes a test for the step name per prior feedback. The changes move force merge logic from a centralized service pattern to a more distributed step-based pattern, where each step handles its own force merge execution through the DlmStepContext deduplication framework.

Changes:

Added maybeForceMerge and forceMerge helper methods to ForceMergeStep for executing deduplicated force merge operations
Made ForceMergeRequestWrapper class and constructor public to allow usage from ForceMergeStep
Removed extraneous testStepName test and added comprehensive tests for the new force merge methods

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
ForceMergeStep.java	Added helper methods (maybeForceMerge, forceMerge, formForceMergeRequest) to handle deduplicated force merge execution with error handling and logging
DataStreamLifecycleService.java	Changed ForceMergeRequestWrapper visibility from package-private to public to enable usage outside the service class
ForceMergeStepTests.java	Removed testStepName test and added comprehensive tests for the new force merge functionality including success, failure, and error recording scenarios

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java

...test/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStepTests.java

...treams/src/main/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleService.java

elasticsearchmachine · 2026-02-27T04:34:40Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

...treams/src/main/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleService.java

...test/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStepTests.java

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java

…ms/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Luke Whiting <luke.whiting@elastic.co>

…Merge()

seanzatzdev · 2026-02-27T20:21:36Z

Decided to just add the execute() implementation to this one since it was just a line and this PR is still short

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java

...test/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStepTests.java

…ms/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Luke Whiting <luke.whiting@elastic.co>

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java

...treams/src/main/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleService.java

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java

…ms/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

dakrone

I left a comment about handling unhealthy indices.

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java

dakrone

LGTM, thanks for iterating on this Sean!

* Add DLM force merge operation helpers * adapt data lifecycleservice force merge code * Update modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Luke Whiting <luke.whiting@elastic.co> * PR feedback * [CI] Auto commit changes from spotless * implement execute(), add completeness check before running maybeForceMerge() * add unit test for when force merge is already complete * add check for nonexistent index * update comment * Update modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Luke Whiting <luke.whiting@elastic.co> * [CI] Auto commit changes from spotless * Update modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com> * respond to PR feedback * use unavailable shards method * address PR feedback * adjust to max timeout for force merge --------- Co-authored-by: Luke Whiting <luke.whiting@elastic.co> Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Add DLM force merge operation helpers

31d6c0d

seanzatzdev requested a review from Copilot February 27, 2026 04:04

elasticsearchmachine added the v9.4.0 label Feb 27, 2026

seanzatzdev added >non-issue :StorageEngine/Data streams Data streams and their lifecycles Team:StorageEngine v9.4.0 and removed v9.4.0 labels Feb 27, 2026

Copilot started reviewing on behalf of seanzatzdev February 27, 2026 04:05 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

adapt data lifecycleservice force merge code

2731ae7

seanzatzdev requested review from dakrone and lukewhiting February 27, 2026 04:33

seanzatzdev marked this pull request as ready for review February 27, 2026 04:34

lukewhiting reviewed Feb 27, 2026

View reviewed changes

seanzatzdev and others added 5 commits February 27, 2026 11:11

Update modules/data-streams/src/main/java/org/elasticsearch/datastrea…

9d0a1c8

…ms/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Luke Whiting <luke.whiting@elastic.co>

PR feedback

69aa5c4

[CI] Auto commit changes from spotless

a1c64ef

Merge branch 'main' into dlm-force-merge-operation

ec18cd9

implement execute(), add completeness check before running maybeForce…

1b45498

…Merge()

add unit test for when force merge is already complete

6b8c255

seanzatzdev commented Feb 27, 2026

View reviewed changes

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java Outdated Show resolved Hide resolved

seanzatzdev added 2 commits February 27, 2026 15:36

add check for nonexistent index

147b93c

update comment

a4e3443

lukewhiting reviewed Mar 2, 2026

View reviewed changes

seanzatzdev and others added 2 commits March 2, 2026 11:07

Update modules/data-streams/src/main/java/org/elasticsearch/datastrea…

ba18299

…ms/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Luke Whiting <luke.whiting@elastic.co>

[CI] Auto commit changes from spotless

a2783d1

dakrone reviewed Mar 2, 2026

View reviewed changes

seanzatzdev and others added 5 commits March 2, 2026 11:38

Update modules/data-streams/src/main/java/org/elasticsearch/datastrea…

def11f4

…ms/lifecycle/transitions/steps/ForceMergeStep.java Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Merge branch 'elastic:main' into dlm-force-merge-operation

aba3ac9

respond to PR feedback

da58897

Merge branch 'elastic:main' into dlm-force-merge-operation

8024964

use unavailable shards method

a155873

seanzatzdev requested review from dakrone and lukewhiting March 3, 2026 20:25

dakrone requested changes Mar 3, 2026

View reviewed changes

address PR feedback

9e83a2c

seanzatzdev requested a review from dakrone March 4, 2026 20:55

Merge branch 'main' into dlm-force-merge-operation

7c46f92

dakrone reviewed Mar 5, 2026

View reviewed changes

.../src/main/java/org/elasticsearch/datastreams/lifecycle/transitions/steps/ForceMergeStep.java Outdated Show resolved Hide resolved

adjust to max timeout for force merge

46f1778

dakrone approved these changes Mar 5, 2026

View reviewed changes

seanzatzdev added 2 commits March 5, 2026 12:07

Merge branch 'main' into dlm-force-merge-operation

64cb76b

Merge branch 'main' into dlm-force-merge-operation

2328dd6

seanzatzdev merged commit 294a8b0 into elastic:main Mar 6, 2026
34 checks passed

seanzatzdev deleted the dlm-force-merge-operation branch March 6, 2026 05:20

prwhelan mentioned this pull request Mar 6, 2026

[ML] Wait for cluster state in test #143767

Merged

prwhelan mentioned this pull request Mar 9, 2026

[Transform] Disable PIT for CPS #143876

Closed

Conversation

seanzatzdev commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seanzatzdev commented Feb 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

seanzatzdev commented Feb 27, 2026 •

edited

Loading